DOCUMENT RESUME 



ED 045 693 



TM 000 15 



AUTHOR 

TITLE 

INSTITUTION 
PUB DATE 
NOTE 



Jung, Steven M . 

Evaluative Uses of Unconventional Measurement 
Techniques in an Educational System. 

American Institutes for Research, Palo Alto, Calif. 
Sep 70 

18p.; From symposium "PLAN in Operation - A Summary 
of Four years Experience in the Evolution of an 
Educational System," at the American Psycholcgica 1 
Association Convention, Miami, Florida, September 
1970 



EDES PRICE 
DESCRIPTORS 



IDENTIFIERS 



EDRS Price MF-SC.25 HC-$1.00 

♦Behavior Development, Behavior Patterns, Community 
Problems, Educational Objectives, Educational 
Programs, *Evaluation Methods, Grade 5, Grade 6, 
Grade 7, Grade 9, Independent Study, ^Individual 
Development, Individualized Instruction, 
♦Measurement Techniques, Self Directed Classrooms, 
♦Student Motivation 

Critical Incident Technique, ♦Project PLAN 



AESTRACT 



The present paper reports on resul+s generated by an 
unconventional application of the testing paradigm. This application 
was made necessary by the nature of several goals of Project PLAN in 
the area of student personal and social behavior, self-planning 
skills for occupational and personal development, etc. Attainment cf 
such goals is very difficult to measure using standardized testing 
procedures. The critical incident technique was utilized to identify 
important aspects of student behavior which were felt to represent 
positive examplars of these goals. From this basis, student 
self-report procedures were developed. These procedures were applied 
to situations which were structured according to cues obtained from 
the original critical incidents. The weight of the findings supports 
the tentative conclusion that PLAN students are making good progress. 
However, the study cannot be considered definitive because of certain 
limitations. A critical incident report form (teacher completed) , the 
teacher's directions for the independent learning and community 
service activities exercises, the criteria for tabulating responses 
to these exercises, and a list of the school classes forming the 
sample are appended. (Author/LR) 
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EVALUATIVE USES OF UNCONVENTIONAL MEASUREMENT 
TECHNIQUES IN AN EDUCATIONAL SYSTEM 
Steven M. Jung 

American Institutes for Research 
ABSTRACT 



The present paper reports on results generated by an unconventional 
application of the testing paradigm. This application was made necessary 
by the nature of several goals of PLAN in the area of student personal and 
social behavior, self-planning skills for occupational and personal develop- 
ment, etc. Attainment of such goals is very difficult to measure using 
standardized testing procedures. The critical incident technique was util- 
ized to identify important aspects of student behavior which were felt to 
represent positive exemplars of these goals. From this basis, student self- 
report procedures were developed. These procedures were applied to situations 
which were structured according to cues obtained from the original critical 
incidents. The results indicated the degree to which the goals had been 
obtained, both in an absolute sense and in comparison with the results of a 
more conventional educational program. 




EVALUATIVE USES OF UNCONVENTIONAL MEASUREMENT 
TECHNIQUES IN AN EDUCATIONAL SYSTEM 



During the late Fall of 1969, persistent rumors circulated through the 
Project PLAN headquarters in F a !o Alto. The essence of these rumors, emma- 
nating from classrooms in which instruction had been individualized through 
the use of this educational system, suggested that students were making prc ~ 
gress toward some of PLAN*s most important educational goals. The mission, 
which we decided to accept, was to gather some hard data regarding these 
rumors . 

Individualized education is thought to have a salutory effect on 
learners which goes beyond the immediate acquisition of facts and skills 
(Bolvin and Glaser, 1968; Flanagan, 1970). These effects usually are refer- 
enced in non-operationalized terms such as self -management , self-responsibility 
initiative, resourcefulness, etc. They are generally thought to come about as 
a result of the individualized system, in which the student is allowed to 
utilize a certain amount of self-direction acquired perhaps through the 
nature of the instruction or through peer modeling. The successful exper- 
iences generated by these attempts at self-direction are presumed to rein- 
force and maintain such characteristics. 

Project PLAN is not unlike other individualized educational systems in 
this regard. Three relevant PLAN goals for participating students are as 
follows : 

(1) PLAN will assist students to take responsibility for their individ- 
ual development ; 

(2) PLAN will assist students to plan , manage , and carry out out their 
individual development; and 

(3) PLAN will assist students to acquire good patterns of social and 
civic behavior — -leadership , i nitiative , integrit y, resourcefulness . 

Wright (1970) has presented a complete list of the goals for students 
in PLAN, stated at roughly the same level of specificity - which is to say 
that these are not ’’behavioral 11 objectives. They are, however, valid educa- 
tional objectives, not unlike some of those posited by an early attempt to 
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define quality education (QUEPS, 1968). 

Such goals are not of vital consequence, however, if (1) there is no 
possibility of structuring the educational experience so that they might be 
translated into some form of student behavior, and (2) if there is no way to 
discover this behavior and make it a matter of public record. It has already 
been noted above and mentioned in this symposium that PLAN has been designed 
to require student behaviors that can be classified as interim steps toward 
such goal attainment. The nature of the teacher training program, the guid- 
ance program, the instructional management system, and even the very structure 
of the teaching learning units (TLU’s) themselves are designed to require 
participation of each student in the educational process. 

These factors of educational system design are adequately covered else- 
where. The thrust of the present problem was to gather evidence that students 
in PLAN were progressing beyond the interim steps required by the educational 
system and were beginning to independently exhibit evidence that they were 
indeed self-directed. Teachers in PLAN schools were fond of saying that they 
could tell a PLAN student in the dark, or blindfolded, because the students 
were so self-directed. Thus, teachers were the first basic source tapped 
in attempting to empirically verify these claims. 

METHOD 

Critical Incident Study 

The method of critical incidents (Flanagan, 1954) was utilized to collect 
from PLAN teachers obseirved events having special significance and relating 
to the areas of self-responsibility, self-management, and resourcefulness. 
Teachers were asked to think of a recent occasion when they observed a student 
exhibit an unusual degree of one of the above traits. They were then asked 
to recount the circumstances surrounding the event and to tell exactly what 
the student did that indicated growth and development with respect to these 
qualities. Finally, they were asked to indicate why they thought the incident 
was a result of the student’s experience in PLAN. 

To collect these data, Critical Incident forms (appendix A) were sent 
to all PLAN teachers (N=374) along with stamped return envelopes. In all, 

56 incidents were returned to AIR in Palo Alto. These were analyzed and 
reduced to the following six statements which best summarize the incidents 
reported: 

(1) started assigned tasks promptly without reminder 
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(2) did unusually thorough job on assigned task or continued beyond 
requirements of assigned or agreed-upon task 

(3) used AV media or books more than required by assigned or agreed- 
upon task 

(4) carried out unassigned learning task without reminder or support 
from others 

(5) completed an unassigned civic, social, or playground activity 
without reminder or support from others 

(6) planned and complete d a strategy or a schedule involving several 
tasks 

In general, the behaviors reported here were cases where students carried 
through on a task or project on their own, with no prodding or reminder. The 
motivation was presumably intrinsic , ie., the student was motivated to per- 
form to please his own interests, to achieve his own satisfactions, rather 
than to satisfy the requirements imposed by teacher, parent, or other authority 
persons. 

The next step was to obtain hard data on the extent to which PLAN students 
in general exhibited such behaviors in comparison with students who were enrolled 
in more conventional educational programs. 
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Community Problem s Exercise 

Within the time constraints imposed by the rapidly approaching end of 
the school year, it was obvious that no massive collection of observations 
in either a natural or a contrived setting was possible. It was decided that 
a student self-report type of exercise was the only practical alternative. 

Such data have proven to have considerable accuracy when the student is asked 
to respond to questions about public events for which some sort of indepen- 
dent verification is possible (Flanagan and Jung, 1970), and they have the 
advantage of being comparatively easy to collect. Forms were therefore pre- 
pared for grade levels 5,6,7,9,10 and 11 (PLAN for grade levels 8 and 12 will 
not be ready for tryout in the schools until this year). 

The forms contained s_ sees for students to report two types of activities 
which it was thought would relate to the critical incidents reported previously. 
These were (1) Independent Learning Activities , including those relating to 
general educational development but not part of the assigned schoolwork; and 
(2) Community Service Activities, including things related to the improve- 
ment of the community. Students were asked to report only activities they 
had performed during the course of the last semester, in order to delimit 
the reporting period and try to keep it within their memory capacity. They 
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were also asked to indicate their role in the organization and planning of 
these activities. Each form contained the following basic instructions: 

List the activities in which you have participated in the past semester. 

These should be activities which were undertaken voluntarily by you. 

For each activity you list, you should indicate your role in the organizing 

and planning of the activity. Use the back of this page if necessary. 

The teacher instructions, part of which were to be read to the class, 
are included as Appendix B. These instructions attempted to: (1) provide 

enough information to teachers about the purpose of the study to gain their 
cooperation; and (2) to standardize the conditions of administration such that 
results would be comparable across classes. As will be noted later, only par- 
tial success was achieved in this effort. The instructions indicated that 
the students should not be given teacher-constructed examples which would 
influence the responses of the class as a whole, since spontaneous and thought- 
ful replies were of most interest. However, it was realized that fifth and 
sixth grade students especially might have difficulty in grasping the concepts 
involved in "independent learning" and "community service" activities. There- 
fore, several examples were given which were taken directly from the reported 
critical incidents. Allowance was then made in the scoring for responses 
which were obviously just repetitions of these examples. 

Materials were sent to the PLAN and control classes via PLAN consultants 
about May 7. Control classes had been selected at the outset of PLAN to serve 
in comparisons such as the present one. In general, these consisted of non- 
PLAN classes in the same school or of classes in neighboring PLAN schools 
judged to exhibit a similar student population. While strict random assignment 
of subjects to classes was not achieved, it is felt that the situation was 
the best that could have been obtained under the circumstances of such a large 
developmental undertaking as PLAN. Excercises were returned from 54 classes, 
of which 37 were PLAN and 17 were control. 



RESULTS 




The number of Independent Learning activities and Community Service 
activities were tabulated for each respondent (N=1400) according to the 
criteria pLesented in Appendix C. In general these criteria were designed 
to insure, to as great an extent as possible, that tabulated activities 
were: (1) learning and community service related; (2) independently under- 

taken; and (3) not the result of some idea implanted solely as a result of 
the directions or other instructions by the teacher. 
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In order to check the adequacy of these criteria for achieving reliability 
in tabulating activities, two persons independently tabulated a randomly selected 
sample of six classes. Table I demonstrates the high level of agreement which 
was achieved. 

(Insert Table I here) 

In order to generate meaningful comparisons in terms of the tabulations 
of these data, only PLAN data were selected for further analysis for which 
appropriate control data also existed. This meant that all analyses were made 
by comparing the data generated by students in PLAN classes with data generated 
by students in other classes in the same school or in a neighboring school. 

This resulted in the selection of 14 (7 PLAN, 7 control) clasres at the 5th 
grade level (PLAN N=231, control N=182), 8 (4 PLAN, 4 control) classes at the 
6th grade level (PLAN N=110 , control N=122) , 2 (one of each) classes at the 
7th grade level (PLAN N=24, control N=31) , and 4 (two of each) classes at the 
9th grade level (PLAN N~60 , control N~52) . These classes and their location 
are indicated in Appendix D. 

The results of the basic tabulations of data by designated category and 
by grade level are presented in Table II. 

(Insert Table II here) 

The total tabulations, corrected on a percentage basis for differences in the 
N’s, are presented in Table III, 

(Insert Table III here) 

In order to attribute a probability statement to the differences which are 
apparent in these tables, Mann-Whitney U tests were performed using the ranked 
frequencies of both Independent Learning and Community Service activities for 
each grade level. The results of these analyses are presented in Table IV. 

(Insert Table IV here) 

The differences at grades 5 and 6 were significant and in favor of the PLAN classes. 

DISCUSSION 

The present study cannot be considered definitive for several reasons; 
however the weight of the findings supports the tentative conclusion that PLAN 
students are indeed making good progress toward the attainment of some of the 
most important objectives of any educational, program. , Further, these data were 
collected in a relatively short period of time with a minimum of expense, pointing 
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out an innovative application of the critical incident technique in evaluating 
a comprehensive educational program. 

The following limitations of the present study are pointed out as a 
cautionary note against overgeneralizing from the results. First, it is 
realized that the limited percent of return for the community problems exer- 
cises implies a selected sample. The fact that over twice as many PLAN as 
control teachers returned exercises is also an indication of this selection. 
The class matching procedures were adopted in an attempt to reduce this source 
of error. Second, the responses from some classes strongly implied that 
students had been given instructions which went beyond those which were to 
have been read. The scoring procedures that were used attempted to reduce 
this error. In order to correct both of these drawbacks, however, much 
more control over the testing situation and much more time for following up 
on non-respondents would be required than was available in the present study. 

To the extent that PLAN and control students were not chosen by random 
selection procedures, the results of the non-par ame trie statistical analysis 
are in question. At present, there is no way to estimate the effects of this 
deficit. 
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TABLE I 



Data on Reliability Check for Tabulating 
Independent Learning and Community Service Activities 



PI-AN Classes 
(N=3) 

Rater 1 Rater 2 



Control Classes 
(N=3) 

Rater 1 Rater 2 
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Comparison of Responses Between 
PLAN and Control Classes by Category 
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TABLE III 



Comparison of Total Responses Between 

PLAN and Control Classes 
(Frequencies corrected for differences in N) 





PLAN 


Control 




N=231 


N=182 


Independent 

Learning 


98 


51 


Community 

Service 


239 


168 




Grade 

N=ilO 


6 

N=122_ . 


Independent 

Learning 


72 


44 


Community 
Servi ce 


113 


83 




Grade 

N=24 
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N=31 


Independent 

Learning 


20 
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Community 

Service 


26 


16 




Grade 

3=60 
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N=52 


Independent 

Learning 
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27 
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60 
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TABLE IV 



Results of Mann-Whi tney U Analysis^ 
for Ranked Frequencies of Independent 
Learning and Community Service Activities 



IL CS 

z U 



Grade 5 


17,518 


-2.97* 


17,707 


Grade 6 


5120 


3.49* 


5251 


Grade 7 


283 


1.50 


318 


Grade 9 


1248 


-1.82 


1361 



* p< .05, 2 tailed test 
1 correction for ties omitted 
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3.24* 

.91 

■1.16 



APPENDIX A 



Think of a recent occasion when you observed a student exhibit an unusual 
amount of some quality such as resourcefulness, independence, initiative, 
self-confidence, responsibility, social sensitivity, etc., which appeared 
to result from the PLAN approach to education. 

In what grade was the student? 

Was the student male or female? (circle one) 

When and where did the incident occur? 



Just what did the student do that indicated desirable growth and 
development with respect to such qualities as those mentioned above? 



Why do you believe this incident is a result of the student's educa- 
tional experience in PLAN? 



APPENDIX B 



Teacher's Guide 
Community Problems Exercise 

General Information 

American Institutes for Research is conducting this survey for the 
purpose of examining the degree of awareness and involvement of American 
students. We are particularly interested in their skills in identifying prob- 
lems and in formulating and carrying out plans of action in areas which are 
of interest to them. A selected sample of schools around the nation is 
participating in this survey. We hope that it will be possible for you to take 
about 45 minutes to implement the survey in your class and to return it to 
us. The results will be returned to you as soon as they can be prepared. 

Instructions 

To administer the exercise, read the following paragraph: 

"Today you are asked to take a few minutes to participate in a survey 
designed to examine the current degree of involvement of American students 
in community problems. This class has been chosen to be representative of 
many students in this area at this grade level. The results of this survey 
will be examined as an indication of the activities and concerns of students 
around the country. We ask, therefore, that you be frank and thorough in 
responding to the following exercise." 

Pass out one "Community Problems Exercise" booklet to each student. 
Ask the students to print their name, school, and your name at the top of 
the booklet in the spaces indicated. Then say the following: 

"On the first page of the booklet, marked 'Activities List, ' you are 
asked to list some of the independent activities on which you have worked in 
the past semester. This includes those which deal with community service, 
improvement of social relationships in or outside of school, exercise of 
your civic responsibilities, etc. These should generally be activities which 
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were undertaken voluntarily by you, rather than assigned. For each activity 
which you list, you should indicate your role in its organization and planning. 
Under the heading ’Independent Learning activities, ’ list things you have 
done in the past semfister that are related to your general development and 
education, but were not part of your assigned schcolwork. This might be a 
science fair project or simply a book you have read which is of interest to 
you. Under the heading Community Service activities, 1 list things you have 
done in the past semester for the improvement of your community. This 
would include such things as starting or helping with a charity drive in your 
school or hometown, writing a letter to the editor of your local paper, or 
similar activity. Try to remember and to list those things you have done this 
past semester and tell whether you planned, organized, or participated in 
each. M 

Allow enough time for all students to finish; then ask them to turn to 
the "Community Problem 1 ’ on page 3. Read the directions which are printed 
at the top of the page. You should then allow exactly 30 minutes for the 
students to complete the exercise. You may rephrase or amplify the 
exercise question if you feel this is necessary, but refrain from offering 
examples which might influence the responses of the entire class. We are 
interested in spontaneous and thoughtful replies. If some students need 
more space for their responses than is provided on the booklet, provide 
them with scratch paper and staple it to the back of the booklet. 

When 30 minutes have expired, gather all returned problems 
exercise booklets (one from each student) and return them to the local 
coordinator. 

Address any questions or comments to: 

Dr. Steven M. Jung 

American Institutes for Research 

P.O. Box 1113 

Palo Alto, California 94302 
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APPENDIX C 



Criteria for Response 
Tabulation by Category 

Independent Learning Activities 

(1) Extra non-required reading or studying 

Scored only when child indicated that he did extra credit reading 
for a purpose; ie., interest in a subject, extra credit book reports, 
enlarge vocabulary. Statement that "I read books” not adequate; must 
show that child took the initiative in activity for educational purpose. 

(2) Research - library, tapes, audio-visual 

Scored when child stated that he sought information on his own - either 
for personal interest or for assignment - but must indicate that source 
used was thought of by child, not assigned or suggested. Includes 
use of library, encyclopedias, reference books, tapes, A-V material. 

Also includes personal educational ,f field trips" to study plants, 
animals, etc. 

(3) Knowledge of needing and/or desiring information 

Scored responses when child actively sought help with schoolwork or 
desired information about a subject of interest to him. Includes 
activities such as attending lectures, group meetings, writing letters 
(to persons, organizations) to obtain information. 

(4) Planned or had major role in an activity 

Scored only when child indicated having organized (or played major 
role) in such activities as a play, panel discussion, club, bulletin 
board, etc. Also included roles as yearbook, newspaper editor, active 
student government member, etc. 

(5) Won prize in contest related to learning 

Science fairs, art; music contests, etc. (Self-explanatory) 

(6) Voluntary membership in academic organizations 

Science club, language clubs, etc. Does not include band, choir, 
cheerleading , etc. (Self-explanatory) 

(7) Voluntary summer school (Self-explanatory) 

(8) Projects 

Scored when child indicated that he initiated or was involved in 
extra-curricular, voluntary project. Included those undertaken 
completely by own self (raising animals, conducting an experiment) 
and submitting a project for an organized program. 



Community Service Activities 



(1) Organized community service activities 

Scored when child indicated voluntary participation in a community 
service activity sponsored or promoted by a group or organization. 
The activity mentioned must be specific (membership not adequate) 
and includes church, scouts and fund-raising projects as well as 
safety patrol, organized tutoring, etc. 



(2) Service activity undertaken by self 

Scored when child listed a service activity performed on his own, 
outside the sphere of groups to which he belonged. Examples of 
this behavior are picking up trash, paper, etc., performine chari- 
table services (ie., donations, visiting old and underprivileged 
groups) participating in political and other types of campaigns 
(passing out leaflets, taking surveys). 

(3) Helped someone 

Scored when specific incident of helping either another person or 
group was reported. This includes helping another child with home- 
work, supervising younger children in play, helping older person 
with yardwork, chores. Incident must be one undertaken for the pur- 
pose of being of service, not performed because it was a regular 
duty or for reward or pay. 

(4) Acted in specific responsible position 

Includes responses indicating that child voluntarily held (and 
generally had earned) a unique position that was service oriented. 
Examples are audio-visual monitor, teacher* s aide, leader roles in 
church activities and other projects. 



Activities which were reported substantially, in the same form by at least 
one-half of a class were considered to be teacher (or otherwise) implanted 
and were not tabulated. 



APPENDIX D 



Listing of Utilized Classes 





PLAN 




CONTROL 




5 th 


Tavennerville Elem 

Wood County Schools, W.Va. 


N=38 


Fairplains (2 classes) 

Wood County Schools, W.Va. 


N=45 




Laurelwood Elem (2 classes) 
Santa Clara Unified 


N=50 


Laurelwood Elem 
Santa Clara Unified 


N=29 




Emerson Elem 

Wood County Schools, W.Va. 


N=29 


Emerson Elem 

Wood County Schools, W.Va. 


N=29 




Reed Elem 

San Jose Unified 


N=31 


Reed Elem 

San Jose Unified 


N=28 




Millard Elem 
Fremont Unified 


N=50 


Millard Elem 
Fremont Unified 


N=21 




Harrison Park Elem 
Penn-Trafford District, Pa. 


N=33 


Trafford Elem 
Penn-Trafford District, Pa 


N=30 


6th 


Mirassou Elem 

Union District, Los Gatos 


N=21 


Mirassou Elem 

Union District, Los Gatos 


N=20 




Reed Elem 

San Jose Unified 


N=30 


Reed Elem 

San Jose Unified 


N=29 




Immaculate Heart of Mary 
Archdiocese of San Francisco 


N=30 


St. Charles 
San Carlos 


N=38 




Harrison Park Elem 
Penn-Trafford District, Pa. 


N=29 


Trafford Elem 
Penn-Trafford District, Pa 


N=35 


7 th 


John Muir 

San Jose Unified 


N=24 


St, Charles 
San Carlos 


N-31 


9th 


Penn High 

Penn-Trafford , Pa. 


N=30 


Trafford High 
Penn-Trafford, Pa. 


N=24 




Franklin Jr. High 
Wood County, W.Va. 


N«30 


Franklin Jr. High 
Wood County, W.Va. 


N=28 



O 




Schools, unless otherwise noted 



are in California 



